Search success and expectations with a computer interface

People’s expectations for success can affect their use of information sources in a variety of ways, including their willingness to search at all, their satisfaction (or frustration) with the success that they encounter, and their confidence in the completeness of their search for specific items. An earlier study using a pencil-and-paper format found people to be overconfident in their ability to locate various items in two entry-level menus to The Statistical Abstract of the United States. In addition, their performance was considerably better with a broad menu, comprised of the 33 chapters in the Abstract, than with a narrow one, comprised of 8 superordinate categories. The present study transferred this task to a computer-interactive format. Surprisingly, neither the transfer itself nor the introduction of performance feedback affected the realism of subjects’ expectations. A review of the two studies, involving 481 subjects in all, in the context of the general psychological literature on confidence assessment, provides some suggestions regarding the design of interfaces for computerized data bases. As computers permeate the home and workplace, the cognitive skills needed to use them become increasingly important. Some of those skills are relatively novel, such as developing mental models of how diverse system components interact when following complex formalized procedures (e.g. [l-3]). Others are refinements of everyday tasks, such as the need to think in precise Boolean terms to query computerized data bases (e.g. [4-61). For still others, the tasks remain the same, but the skills need to be exercised in a new substantive domain, such as learning the names of objects and procedures that have been labeled by someone from a different subculture (e.g. [7-g]). One familiar skill that needs to be exercised regularly with computerized systems is evaluating the chances that a procedure will be successful. On the basis of such expectations, users can determine whether to continue with the procedure, whether to ask for help, how much to invest in the effort, and how to plan for possible failures. As Bookstein [lo] notes in the specific context of computerized data bases, “Uncertainty and incompleteness [are] intrinsic to both indexing and retrieval” (p. 122). Without a realistic assessment of uncertainties, users may overestimate their chances of success, leading to unwise investment of resources, premature frustration with systems that have failed to fulfill promises that they never made, uncritical acceptance of incomplete or erroneous system products, and inadequate attention to the sort of premonitors of failure that could be exploited to improve subsequent usage. Underestimating one’s chances of success can lead to complementary difficulties. Cognitive psychology has devoted considerable attention to the determinants and appropriateness of people’s expectations [ 1 l-151. This research typically finds that people have no difficulty expressing their confidence in succeeding at a task in quantitative terms (e.g., odds, probabilities), that people tend to be more knowledgeable when they are more confident, but that their absolute level of confidence is not a reliable indicator of their absolute level of knowledge. Figure 1 shows one very common pattern of results. As confidence (indicated by judged probability of success) increases, so does knowledge. However, the rate of change *This research was supported by the National Science Foundation under Grant SES 8312482 to Perceptronics, Inc. Any opinions, findings and conclusions or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of the National Science Foundation. Our thanks to RickReed, Mark Layman, Leisha Sanders, and Sonny Eberts for help in various stages of this process.